Deconvolving far-field images using scanned probe data

ABSTRACT

A method for deconvolving far-field optical images beyond the diffraction limit includes the use of near-field optical and other scanned probe imaging data to provide powerful and new constraints for the deconvolution of far-field data sets. Near-field data, such as that which can be obtained from atomic force microscopy on a region of the far-field data set in an integrated and inter-digitate way, is used to produce resolutions beyond the diffraction limit of the lens that is being used. In the case of non-linear optical imaging or other microscopies, resolutions beyond that which is achievable with these microscopies can be obtained.

I. FIELD OF THE INVENTION

The field of the invention is the combination of scanned probemicroscopic data with far field optical and other images in order todeconvolve these images beyond the diffraction limit.

II. BACKGROUND OF THE INVENTION

Lens based far-field imaging is limited in the resolution that it canachieve by the characteristics of the lens. In general, there areproblems of diffraction of the lens, problems with aberration of thelens and problems of out-of-focus radiation. The latter, out-of-focusradiation problem is generally partially improved by the use of confocalimaging methodologies; in optics, non-linear imaging techniques are alsouseful. The solution of the former diffraction and aberration problemsis partially addressed by measuring the point spread function of thelens and then using computer deconvolution to remove these effects fromthe image. Even the latter out-of-focus problem can be addressed withoutconfocal or non-linear imaging by considering both the in-focus and theout-of-focus point spread function and using deconvolution routines totry and eliminate these effects. Numerous algorithms have been devisedto address these problems of computer deconvolution of far-field imagingdata, but none are completely successful and none of them have theability to carry the far-field image to the realm beyond the diffractionlimit as defined, for example, by the Rayleigh criterion, which isapproximately ½ of the wavelength of the radiation that is being used.For visible 500 nm light this is 250 nm.

In terms of deconvolution algorithms, a powerful mathematical approachis based on the use of constraints. For example, in deconvolving afar-field image a good constraint would be to define with high precisionthe cell membrane of a cell that is stained with a dye and is beingimaged by a lens. By precisely defining the position of a cell membraneor a portion of the cell membrane, it is possible precisely to definewhere the staining in the image is confined and beyond which point orpoints there is no staining and its associated optical phenomenon. Sucha constraint would give many deconvolution algorithms a powerfuladvantage. Nonetheless, even though the idea is mathematically apowerful concept [Carrington et al. Science 268, 1483 (1995)], it isseriously limited in far-field optics by the inability to obtain aconstraint that is better than the optical resolution.

III. STATE OF PRIOR ART

No one has previously attempted to incorporate near-field optical dataand other scanned probe microscope data such as that which is obtainedfrom atomic force microscopy to the problem of providing constraints inthe deconvolution of far-field optical and other far-field imagingtechniques.

IV. SUMMARY OF THE INVENTION

In accordance with the present invention, a method for deconvolvingfar-field optical images beyond the diffraction limit includes the useof near-field optical and other scanned probe imaging data to providepowerful and new constraints for the deconvolution of far-field datasets. Near-field data, such as that which can be obtained from atomicforce microscopy on a region of the far-field data set in an integratedand inter-digitate way, is used to produce resolutions beyond thediffraction limit of the lens that is being used. In the case ofnon-linear optical imaging or other microscopies, resolutions beyondthat which is achievable with these microscopies can be obtained.

V. BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing, and additional objects, features and advantages of thepresent invention will become apparent to those of skill in the art fromthe following detailed description of a preferred embodiment thereof,taken with the accompanying drawings, in which:

FIG. 1A illustrates a first image of a model object, a second image ofthe object imaged by a lens having a known point spread function, athird, or sampled image recorded by a CCD device, and a fourth image ofa deconvolution of the sampled image without constraints;

FIG. 1B is a graphical illustration of the deconvolution of FIG. 1A;

FIG. 2A illustrates the same images as FIG. 1A, wherein thedeconvolution is performed with constraints obtained from near-fieldimaging data;

FIG. 2B is a graphical illustration of the deconvolution of FIG. 2A; and

FIG. 3 is a diagrammatic illustration of equipment suitable for carryingout the method of the invention.

VI. DESCRIPTION OF THE INVENTION

The present invention incorporates data that has never been incorporatedpreviously to resolve issues and problems in far-field imaging. Thisdata comes from near-field optical microscopy and its scanned probecousins, such as atomic force imaging (AFM), and presents the far-fieldmicroscopist with constraints that will dramatically improve thefar-field imaging of all forms of far-field microscopy. Theseimprovements are available for both linear and non-linear opticalmicroscopy and even those microscopies that use particles rather thanelectromagnetic radiation. The invention also allows for using one formof scanned probe microscopy to deconvolve another, to thereby improvethe resolution of a scanned probe microscope before this data is used inthe present invention, as described below.

In accordance with the invention, it is essential to obtain near-fieldoptical or other scanned probe imaging data in a way that is fullyintegrated with a far-field data set that is to be improved. In oneembodiment of this invention in which far-field optical microscopic datais to be deconvolved, one useful approach to achieving the required fullintegration of the data sets is to use a charge coupled device (CCD) torecord the far-field data that corresponds to an associated scannedprobe pixel. Furthermore, it is additionally useful if the scanned probedata sets that are to be used in the deconvolution are obtained insimultaneous channels. This can be done, for example, with a tip that ismultifunctional such as a tip that is both a subwavelength light sourceand an AFM sensor [K. Lieberman, et al., Rev. Sci. Instr. 67, 3567(1996)] that can be used in contact or near-contact with a surface of aspecimen that is being imaged.

However, even before the far and near-field imaging process is begun,lens images of a subwavelength light source of known dimension arerecorded on a CCD. These images, both in-focus and out-of-focus, areused to obtain a measure of the lens point spread function (PSF), evenwith perturbation of the object, or sample, being imaged. This PSF isthe lens function that convolutes with the functional representation ofa sample to give the blurred image that is the far-field image with itsassociated diffraction and other problems mentioned in the Backgroundsection, above. Alternately, the convolution effect of the lens can alsobe determined if a known high resolution sample is imaged and the errorbetween the real and the ideal image is represented as a blurringfunction introduced by the lens. Obviously, if the imaging task isfluorescence then the high resolution test object will have to besimilarly fluorescent. Thus, the first task in this method is todetermine the PSF of the lens that is to be used to image the desiredobject, or sample.

The next step is to record with the CCD the far-field image of theobject. Subsequently, super-resolution optical data is recorded forspecified points on the object surface. An example of such data caninclude defining an exact point at which the optical contrast in anobject terminates; i.e., defining the edge of an object to much betterthan the optical resolution if the far-field image is an optical imageor defining the x, y and z point, or voxel, at which there is a contrastchange, and relating this point to another point of contrast change inthe sample. The two points of contrast change in the sample could, forexample, be at different planes as defined by the lens in the far-field,and the near-field scanned probe data, obtained through amultifunctional AFM sensor tip and simultaneously recorded, couldprovide not only the x-y separation of the two points but also the Zseparation at a resolution that is better than any optical approach suchas confocal microscopy.

The exemplary constraints listed above, or for that matter anyconstraints from scanned probe technology, have never been used in thiscross-fertilization mode with far-field optical microscopy or, for thatmatter, any far-field microscopic technique. In addition, suchcross-fertilization has not been used between scanned probe techniques,i.e. to use near-field optical data as a constraint at deconvolvingnear-field atomic force microscopy data, or the reverse. With regard tothis latter mode of deconvolution there is quite a bit of synergismbetween, for example, the near-field optical and the near-field AFMsince the functional dependence of the decay of the effect, as afunction of probe sample separation, is near-exponential for thenear-field optical and occurs over a much shorter distance for thesimultaneously recorded AFM technique. In essence, then, the scannedprobe microscopy synergism can first be used to improve the near-fieldscanned probe microscopy data and then that data can be applied as aconstraint to the far-field microscopy deconvolution in question. Atthis point, it is also important to note that the order of theprocedures listed in this section is not a critical part of theinvention; any combination of the order of the steps or partialcombination of steps constitutes this invention. For example, if thenear-field optical data is not used to deconvolve to higher resolutionthe atomic force microscopy data, it, and the atomic force microscopydata could still be used to deconvolve the far-field data.

The CCD mentioned above is a most useful method to obtain the digitizedimage of the far-field, but this can also be accomplished with confocalmicroscopy. In the case of far-field optical microscopy it should benoted that there could be innovative ways to record the confocal datawithout any confocal aperture. For example, a film of material thatproduces a non-linear optical signal known as second harmonic generation(SHG) can be used as part of this invention to record a confocal image.In this case the light from the plane of focus is focused by the lensonto the film, such as a plastic film of purple membrane that producesSHG [Z. Chen et al., Applied Optics 30, S188 (1991)], with an intensitythat is higher than from any other plane in the sample that is beingfocused by the lens. A film that would produce a second harmonic signalonly where there is a point of light from the plane of focus in thesample could be used to replace the confocal pinhole. Such a film wouldact as a parallel filter for light from the plane of focus in thesample. This could be used together with an appropriate filter after thefilm to remove the fundamental wavelength that was illuminating thesample and to pass only the SHG to the detector which could be a CCDrather than the single channel detector that is normally part of aconfocal set-up. This could be done with SHG or other non-linearoptically active films.

VII. ADVANTAGES OVER PRIOR ART

Scanned probe microscopy data has not been used as a constraint inmathematical constraint algorithms to deconvolve far-field opticalimages. In addition, the use from multi functional scanned probemicroscopy of one parameter, such as near-field optical data, todeconvolve another parameter such as atomic force microscopy data hasalso not been applied. The advantage over prior art arises from theincrease in spatial resolution that this approach achieves.

VIII. APPLICATIONS

Methodologies for increased spatial resolution always open new doors inscience and technology, as evidenced by the revolution that was causedby the introduction of the electron microscope.

To test the essence of this invention a calculation has been performedon a model far-field optical data set, as illustrated in FIGS. 1A, 1B,2A and 2B, to which reference is now made. In FIG. 1A there are fourimages, one in each of four horizontal rows. At the right-hand end ofeach row is an intensity legend for comparison. Starting from the top ofFIG. 1A, the first row of the figure represents a model object which hasa dimension that cannot be resolved optically; for example, an objecthaving a dimension of about 0.2 micron. As may be seen in the first row,the object has a first sharp light-to-dark transition point at itsleft-hand edge, a second, dark-to light transition near the center, athird, light-to-dark transition to the right of the second transition,and a fourth, dark-to-light transition at the left-hand edge of themodel object. When the object is imaged by a lens with a known pointspread function, each of the transition points on the object is blurredbecause the dimensions are too small to be resolved, and the blurredimage, which is the second row from the top in FIG. 1, results. Whenthis image is recorded by a CCD there is further blurring due to thepixel character of the CCD, as illustrated in the third row. The lastimage, row 4 in this figure, is produced by processing the image of row3 with a standard deconvolution algorithm without the imposition of thetype of constraints that are central to this invention, in which a newapproach to provide constraints is described. FIG. 1B is a chartillustrating the intensity variations produced by the model object andby the restored (deconvoluted) object.

In FIG. 2A the same object, lens, and CCD are used, and rows 1-4illustrate the same object and images as described above with respect toFIG. 1A, except that in the deconvolution algorithm four points aregiven the resolution of near-field optics. These points are, in goingfrom left to right in the model object illustrated in the top row ofFIG. 2A, the first, second, third and fourth alterations in contrast, asdescribed above with respect to FIG. 1A. The results of using suchconstraints is seen in the vastly improved quality of the deconvolvedimage in the bottom row of that Figure.

Diagrammatically illustrated at 10 in FIG. 3 is equipment suitable forcarrying out the above-described methodology. A near field microscopesuch as a scanned probe microscope 12, includes a near field opticaland/or a near field atomic force microscope, is positioned so that thesame region of an object or sample 14 can be imaged by both themicroscope 12 and a far-field imager 18 which is based on a lens 20, asindicated by the doted lines. The sample 14 has to be put on a stage 16that can be accurately moved relative to the scanned probe or far-fieldimaging devices. Both the scanned probe data and the far-field imager'sdata can be integrated by a processor 22 that can be used eitherdirectly or with another processor so that the recorded data can be usedfor computation of the final image.

The near field microscope can be configured with a probe that is anear-field optical probe. This probe can be positioned at any point onthe sample. One reason for incorporating such a probe would be toprovide an on-line point source for determining a point spread functioneither with or without the sample in place. The relative movement of thesample and the probe allows for multiple point spread functions atdifferent points in the sample where the sample could perturb the pointspread function in different ways. The near-field optical probe in thenear field microscope 12 can not only provide point spread functions butalso other optical information at various points in the sample. Inaddition, the near-field optical probe can be configured so that theheight of these or other points on the surface of the object or samplecan be correlated with the far-field optical image. Furthermore, thelocation of the borders of the sample can also be assessed. Such heightinformation can also be obtained by other types of probes in the nearfield microscope, including an atomic force probe, which can alsoprovide information on the borders of the object or sample.

The processor records and integrates all the data, including the opticalinformation, height information, information on the object or sampleborders and/or accurate movement of the object or the sample relative tothe near field microscope and/or the far field imager. This data,together with any other pertinent imaging information, can beincorporated into a deconvolution algorithm that can then use this datato produce a super-resolution deconvolved image.

As an example of the closed-loop operation of the illustrated equipment,an image is obtained with the far-field imager and the position of aparticular feature is determined with the near-field microscope. Theimages are directed to the processor, where they go through thedeconvolution algorithm, using the portion of that particular feature asa constraint. The deconvolution algorithm may provide calculatedinformation about another feature at another position in the far-fieldimage, and the near-field microscope is then used to measure that newfeature. The measured information is used to determine the errorgradient in the position of the new calculated feature, and in closedloop fashion the algorithm is adjusted to improve the result.

In various embodiments of the invention, the far-field imager mayinclude non-optical linear optical imaging and the imaging and the imagerecording may include recordation of interdigitated and correlated datasets of scanned probe and far-field imaging. It is beneficial but not anabsolute requirement that neither the near field microscope nor thefar-field imaging device obstruct viewing of the object or sample fromone or the other device.

Although the invention has been described in terms of preferredembodiments, it will be understood that numerous variations andmodifications may be made without departing from the tnie spirit andscope thereof as set forth in the following claims.

1. A method for deconvolving far-field optical images for improved imageresolution, comprising: employing near-field microscopy as a pointsource that is positioned to provide images of specific regions of asample corresponding to specific pixels in a far field imager, thenear-field microscopy being movable with respect to a surface of thesample to obtain near-field optical data; recording near-field data forthe relative height of the point source with respect to the samplesurface and near-field data for the location of the borders of thesample; obtaining far-field optical image data corresponding to thesample; recording said far-field optical image data simultaneously withthe data obtained from the near-field optical microscopy; andincorporating the far-field and the near-field optical data indeconvolution algorithms using the data from the near-field imaging foradded precision of the far-field imaging or as a constraint with thedeconvolution algorithms to produce a deconvolved super-resolutionimage.
 2. The method of claim 1, further including obtaining thenear-field optical microscopy data in simultaneous channels.
 3. Themethod of claim 2, further including fully integrating the far-fieldoptical data with the near-field optical data.
 4. The method of claim 1,further including obtaining said far-field optical image data through alens.
 5. The method of claim 4, further including determining apoint-spread function of the lens prior to obtaining said near-field andsaid far-field data.
 6. The method of claim 5, wherein determining thepoint-spread function of the lens includes determining the differencebetween real and ideal images of a sample to determine a blurringfunction of the lens.
 7. The method of claim 6, further includingdetermining the point spread function of the lens with atomic forcetopography information from said sample.
 8. The method of claim 1,wherein employing near-field optical microscopy includes scanning saidsample with subwavelength resolution to define optical contrast pointson said sample.
 9. The method of claim 1, wherein obtaining saidfar-field optical image data includes non-linear optical imaging. 10.The method of claim 1, wherein recording far-field and near-fieldoptical images includes recording interdigitated and correlated sets ofimage data.
 11. The method of claim 1, further including determining adifference between the deconvolved optical image and one or more pointsof the near-field image to compute an error value for directing thecomputation of a newly deconvolved image.
 12. The method of claim 11,wherein the deconvolution is a closed loop in which the error value isminimized.
 13. The method of claim 1, wherein employing near-fieldmicroscopy includes employing a scanned optical probe.
 14. The method ofclaim 1, wherein employing near-field microscopy includes scanned probeimaging combined with atomic force imaging to provide two near-fieldimages.
 15. The method of claim 1, further including combiningnear-field and far-field imaging without obstructing the far-fieldimaging to produce fully integrated data sets for deconvolving theoptical images.